feat(criteria): accept glob patterns in criterion path fields - #65
Merged
Conversation
A criterion `path` is a literal, so a task must hardcode where an artifact lands. When the prompt does not pin that location — a scaffolding tool that creates a wrapper directory the agent names itself — a correct artifact in an unexpected directory scores 0.0 on the path alone, while a sibling criterion that discovers the file reports the run as valid. Resolve `path` through `Sandbox.resolve_files`, which expands a pattern containing `*`, `?`, or `[` against the sandbox root and leaves a literal path untouched. Every path-based criterion inherits it: file_exists, file_contains, file_matches_regex, file_check, json_check, import_check. `file_exists` passes on at least one match. Content reads require exactly one match and otherwise raise, listing every match — an ambiguous pattern is captured as a scored-0.0 result rather than silently grading one arbitrary file. Matches are sorted for determinism and directories are dropped.
jiyangzh
requested review from
akshaylive,
bai-uipath,
tmatup and
uipreliga
as code owners
July 29, 2026 23:02
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
bai-uipath
approved these changes
Jul 30, 2026
bai-uipath
left a comment
bai-uipath
left a comment
Collaborator
There was a problem hiding this comment.
Right seam to put this on, and requiring exactly one match for content reads is the correct call over max/min-over-files, which would invent a scoring rule and let a staged fixture satisfy a check. One thing needs to land before merge; the rest are optional.
Major
The glob runs unfiltered over the sandbox root at grading time. pathlib doesn't skip dotdirs, and the skills post_run prune of node_modules/.venv runs in teardown, after evaluation, so a **/*.json pattern matches vendored files and grades or hard-fails on them; **/*.flow works today only because the extension is rare, which makes this latent rather than absent. Fix: filter the glob branch through the existing get_ignore_patterns/should_ignore_path helpers in the resources module (already used for the template copy), passing the sandbox-relative path since they match on any path component. Worth a line in the guide that dist and build are in that default set, with ignore_patterns: ["!dist"] as the escape hatch.
Fix if you agree, otherwise lgtm
- The follow-up migration should glob away only the segment the prompt leaves free. The failure is an unknown wrapper prefix, not an unknown filename, so
**/<Name>.flowdrops the prefix and stays unique; a blanket**/*.flowturns exactly-one into a hard 0.0 on any task carrying a second flow file. - Nothing records which file was graded. With exactly-one semantics the message is most of the feature: put the resolved path in
detailson the passing path too, and cap the ambiguity list at ~10 with+N moresince it persists into task.json. - Minor: try the literal path first and only fall back to globbing, so a real filename containing
[or?isn't silently reinterpreted;import_checkin the guide isn't a criterion type, whilejson_checkandclassification_matchinherited the behavior and still carry pre-glob field descriptions;reference_comparison's agent file reads the sandbox directly and bypasses the new seam; the sortedness test asserts againstsorted()of its own output.
uipreliga
self-requested a review
August 3, 2026 17:36
jiyangzh
marked this pull request as draft
August 3, 2026 23:58
…tered
Review follow-ups on the glob-in-`path` seam. Two of them changed what a
run scores.
Literal-first resolution. `Path.glob` turns `[...]` into a character class,
so sniffing for `*?[` reinterpreted a plain filename as a pattern: a real
`report[2024].json` resolved to a `report2.json` decoy and was graded
silently, and `logs[1]` — which exists — globbed to nothing, flipping
`file_exists` to 0.0 for unchanged agent output. `resolve_files` now probes
the literal path first and only expands when it does not exist. This is not
just hand-written YAML: dataset fan-out substitutes `${row.<field>}` into
criterion paths, so a row value carrying a metacharacter injected glob
semantics per-row.
Ignore-pattern filtering. Expansion walked the whole sandbox root, which
holds harness-created content the agent never authored — `.venv` (created
inside the root for any task with a `python:` block), copied template trees,
`node_modules` — and pathlib descends into dotdirs. `**/*.json` could pass
off a vendored `package.json`, or hard-fail on ambiguity that has nothing to
do with the agent. Glob matches now filter through the same
`get_ignore_patterns` / `should_ignore_path` used for template copying, on
the sandbox-relative path. A segment the pattern names literally is an
explicit opt-in and survives, so `dist/**/*.js` still grades `dist`;
`ignore_patterns: ["!dist"]` un-ignores a segment a wildcard discovers.
Also:
- `reference_comparison.agent_file` read `sandbox_dir` directly and bypassed
the seam, so path semantics differed per criterion. It now routes through
`get_file_content` like every other path field.
- The graded file is echoed as `resolved: <path>` in criterion details —
with exactly-one semantics, which file was picked is most of the signal.
- The ambiguity error caps its listing at 10 matches with `+N more`; it is
persisted to task.json and injected into judge prompts.
- `json_check.path` / `json_schema`, `classification_match.path` and
`agent_file` carried pre-glob field descriptions.
- The guide listed `import_check`, which is not a criterion type.
- `test_matches_are_sorted` asserted against `sorted()` of its own output.
New lint rule CE032 fails a criterion checker that joins a path onto
`sandbox.sandbox_dir` instead of using the seam — the mechanically
detectable root cause of the `agent_file` drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
akshaylive
marked this pull request as ready for review
August 10, 2026 20:22
|
Claude finished @akshaylive's task in 1m 45s —— View job Review in Progress
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A criterion
pathis a literal, so a task must hardcode where an artifact lands. When the prompt does not pin that location, a correct artifact scores 0.0 on the path alone.Real case from the UiPath/skills smoke gate.
uip maestro flow init <Name>scaffolds a wrapper solution directory whose name the prompt never specifies. The agent choseInvoiceApprovalSolution/, the task assertedInvoiceApproval/InvoiceApproval/InvoiceApproval.flow:run_command— validate the discovered flow"Status": "Valid"file_exists— hardcoded pathfile_contains— hardcoded pathTask scored 0.375 on a valid artifact and failed the ≥95% pass-rate gate. The workaround was a per-repo helper script that globs and shells out — one more thing to maintain, and it only fixes the tasks that adopt it.
Change
pathresolves through a newSandbox.resolve_files. A pattern containing*,?, or[expands against the sandbox root; a literal path is untouched.Because every path-based criterion already routed through
sandbox.file_exists/sandbox.get_file_content, they all inherit this from one change point:file_exists,file_contains,file_matches_regex,file_check,json_check,import_check.Semantics:
file_existspasses on at least one match.ValueErrorlisting every match —ValueErroris not in_ESCALATING_EXCEPTIONS, sohandle_criterion_errorscaptures it as a scored-0.0 result with the message. An ambiguous pattern is reported, never silently resolved to one arbitrary file.pathbehaves exactly as before — no existing task changes meaning.Verification
tests/test_glob_paths_in_file_criteria.py— unit coverage ofresolve_files(literal, glob hit, glob miss, ambiguous, directory-skipping, sort order, unset sandbox) plus end-to-end coverage throughSuccessCheckerforfile_exists/file_contains/file_check, including the ambiguous-glob 0.0-with-message path.ANTHROPIC_API_KEY, Windows-only).ruff format --check,ruff check,pyright(0 errors), and the custom-lint suite (166 passed) all clean.Follow-up in UiPath/skills
Once this releases and the
tests/.coder-eval-versionpin moves off0.8.8, the interim helper script is deleted and ~19 task files with hardcoded<Name>/<Name>/<Name>.flowpaths collapse topath: "**/*.flow".